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Criticizing Instructional Materials Evaluation: 
Adding Meaningful Dimension 



Overview 

Critical evaluation of instructional materials is an important but often neglected aspect 
of instructional preparation. Teachers do not usually select their instructional materials (in 
whatever the format) according to formal, generalized, research-based prescriptions for doing 
so. And only rarely do teachers consider criteria beyond those which are achievement and 
technically oriented (Rothe, 1991). Instead, we are likely to develop and use our personal, 
situated routines to select and evaluate materials. This study suggests evaluation that is more 
theoretically grounded in socio-cultural and local, context-sensitive bases than are many current 
selection/evaluation guides. It reviews the development of evaluation tools, provides new 
measures, and applies new measures to current visual materials. A new evaluation instrument 
was developed and applied to materials for teacher education. From preliminary research in 
this area, we expect the interpretation to reveal a number of explicit and implicit ideological 
stereotypes and assumptions about our beliefs about teachers, teaching, classrooms, and 
mediated education. We believe as our project continues that examining this visual material 
will provide new understanding of our field and our educational beliefs. 

Background and Rationale 

Evaluation of materials has traditionally been derived from behavioral theory and the 
systematic design of instruction so basic to our field. Within these theories, learning is 
conceived narrowly as the learning objectives and the methods used to get students to achieve 
those objectives. The limitations of these theories as a basis for evaluation or selection of 
materials have been repeatedly pointed out by current critics of educational technology. 
Bowe ;s (1988) discusses the "current blindness" of the "conduit" view of software evaluation, 
for example (p. 47). Bowers goes on to critique the instructional procedures of an evaluation 
model based upon Gagne's events of instruction. He points out that such a model ignores the 
subjectivity of language and the awareness of the influence of language on thought. This is 
just one example of criticism which points out the severe limitations of our current evaluation 
models. 

The field of educational communications needs to revisit our theoretical basis for the 
simple models of evaluation or selection which we have been using for so long. Kerr (1985) 
has suggested that methods drawn from sociology, policy sciences, and anthropology could 
"shed new light on problems that have traditionally been approached using psychological 
research methods" (p. 4) Kerr believed that asking new questions in less traditional ways was 
critical to the future of education. With more multi-cultural and critical dimensions of 
education becoming a reality, the area of materials evaluation and selection and the messages 
being given to teachers need to be re-examined. 

Eisner (Eisner and Peshkin, 1990) suggests new methods, and comments on the 
limitations of old models. "The methods into which we have been socialized provide powerful 
filters through which we view the world. If we are predisposed to focus upon the unique 
features of a person or situation and skilled in their literary description, we are likely to attend 
to classrooms in ways that emphasize or make salient the idiosyncratic. If we are inclined 
toward the description of measured relationships or the incidence of events, if what we are 
inclined to think about is how things co-vary, we see the "same" classroom from a very 
different angle and with a very different intent. In short, our methodological concepts 
influence our perception. Thereafter, cognitive maps help us find our way in the territories 
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we wish to explore. Those with different maps tend to take different roads." (p. 9) Eisner 
points out tliat our methods create filters on our perceptions. 

Yeaman (1991) discusses the importance of the socio-cultural aspects of our field, 
wWch he calls neglected. "Educational communications and technology research concentrates 
on the efficiency of learning stimuli, instructional techniques and matching these with student 
characteristics." (p. 990) 

And Ellsworth (Ellsworth and Whatley, 1990) has conmiented that: "Such research 
privileges questions of how individuals process information from educational media over and 
above questions about the ideological work of meaning construction, content selection, setting 
of objectives, and media use. The concep/ of information processing as developed within 
cognitive psychology and applied in educational technology cannot explain meaning, intent and 
significance." (p. 49) 

An examination of evaluation from a socio-cultural viewpoint begins to provide more 
powerfiil and meaningful dimensions in the evaluation of me^ia materials, historical or cutting 
edge. This project begins to integrate media criticism and post-structuralism as a basis for 
improved evaluation, and provides a sample application. The application involves vi.sual 
materials for teacher education: film, video and multi-media. 

Teachers have conceptual and attitudinal knowledge of technology based on their 
experience and the methods and materials used to teach them about the technology. These 
materials remain largely unexplored from both an aesthetic and a critical or formal perspective. 
Likewise, only recently have a few scholars turned their critical attention to educational films 
and videos in order to explain their messages, overt and otherwise. The visual representations 
and the actual construction of knowledge in these materials are now beginning to be 
investigated. Ellsworth's (Ellsworth and Whatley, 1990) research is an example of such work 
and is used to guide the work presented here. 

In their text. The Ideology of Images in Educational Media, (1990), Ellsworth and 
Whatley discuss this recent move to examine the ideology of images. They point out that until 
very recently, educational communications research wa*^ still focused on effectiveness and on 
how learners gain from mediated messages. Recently, educational media researchers have 
turned their attention to educational films and other materials, and are investigating the visual 
representations and the actual construction of knowledge. Ellsworth's study cites examples of 
research examining science and health films and print materials. 

The aim of this multi-phase project is to suggest substantial changes in the ways 
teachers evaluate curricular materials for teaching; to begin an historical record of our field 
in terms of film and video; and to bring a more grounded and broadly literate understanding 
of instructional materials to both theorists and practitioners. 

The results will be summarized and examined for the messages we can discover 
involving materials, teachers, pedagogy, and technology. In addition, this work will focus on 
the hidden curriculum of such messages, describe what was considered valid knowledge, and 
determine who possessed it. Our educational culture will be described, and new ways of 
looking at educational media will be explored. In completing this project we will produce a 
brief history of the teaching of media utilization and evaluation, test a new evaluation tool, and 
analyze several mediated materials. 
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The presentation itself was an introduction to postmodernism and its emergence as an 
important viewpoint for educators (see Suggested Reading List), followed by an evaluation 
activity. Questions raised by a postmodern perspective include issues of emancipation, ethical 
behaviors, and empowermedt. ^In media postmodernism leads us to question who is 
delegitimized and "othered" by mass media such as television. These same questions were 
then raised while reviewing a newer educational videotape from a series. A new evaluation 
form based upon work by Rothe (1991) was shared. This form was prepared using socio- 
cultural analysis aspects to provide several meaningfxil new dimensions to the evaluation 
process (see Instructional Materials Evaluation). Participants were invited to use and critique 
this new instrument, and comment upon its perspective and its usability. Discussion followed 
about the process and the viability of socio-cultural evaluation in schools and by teachers. The 
new form encourages viewers to "see" in a new way; to not ignore important, if subliminal, 
visual aspects. 

This presentation session suggested that typical evaluation forms and models are little 
used because they are severely restricted in their conceptions of learning. Learning is 
conceived narrowly as the learning objectives and the methods used to get students to achieve 
those objectives. Too often, evaluation of instruction and instructional materials does do not 
fully consider the political, economic, language, contextual, and other cultural influences on 
learning. 

Take, for example, the idea of rule learning. A conventional approach usually asks 
us to evaluate materials based on whether or not students learned the rules of, say, written 
English as specified by learning obje<:tives. We want to say that someone has learned 
something more important when he or she has learned those rules and, additionally, 
understands a variety of meanings of using th^j formal rules. A person should know when, in 
what context, she has the option to use one set of languages rules or another. 

These extended understandings of knowing /learning are crucial because they are in 
realms where many, even most, of the important influences on learning reside. We can select 
a material based upon what it explicitly teaches and how it explicitly teaches discrete 
knowledge, but if that learning is politically, ecologically, or economically discouraging to the 
learner and the learner doesn't get a chance to understand the politics, ecology, or economics, 
to that extent learning (and teaching?) is a failuie. For example, if learning is taught as a 
predominantly individual and cognitive activity devoid of any emotion or community, other 
useful ways of knowing are apt to be stunted or dominated for powerful purposes. 

Likewise, a teacher with a restricted idea of materials and with restricted materials 
is less likely to see the changes she or he looks for in the learner, whether those changes are 
task specific (as we say) or are in more general terms of learner attitudes and strategies toward 
learning and knowing. 

An evaluation form such as the standard one reproduced in most Instructional 
Technology basic textbooks (ie. Heinich, Molenda, and Russell, 1993) asks teachers to 
examine materials using a very narrow range of issues about learning and appropriateness. 
The materials are evaluated for their use with a particular audience, and it is assumed that the 
only learning which needs to be considered is that of the intended objectives of the material. 
Such a typical form allows no room for cultural analysis; it does not ask teachers to consider 
the cultural ramifications of the materials, nor does it imply in any way that those are issues 
a teacher may want to consider when selecting and using any material for their classroom. 
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A more involved and extended approach to evaluation is one suggested by Rothe 
(1991). Questions regarding language, culture, power, and inclusiveness are included in such 
an approach. While teachers may consider this method more time-consuming, it is actually 
an approach which helps them plan the use of their materials while previewing them. No 
instructional materials should be included in classroom use without some discussion of their 
creation and intent. Teachers need to be encouraged by this evaluation process to mclude 
discussions about the use of langxiage, images, and the representation of culture wluich any 
materials represent. Newer technologies, especially, can "other" people, continuing to 
severely limit the meanings of what is learned, and avoid questions of ecological damage done 
by the creation and use of technology. 

Just recommending that teachers reconsider their evaluation of materials is not 
enough, but it is one simple method by which teachers can be encouraged to gain control over 
materials selection, production, utilization, and evaluation for themselves and for their 
students. All participants in the education process need to be encouraged to consider the 
visual, cultural and media literacy issues often avoided by more traditional evaluation 
procedures. The ideas presented here could be a basis upon which to build a more informed 
and involved group activity, seeking support for inclusion of the political, economic and 
cultural issues within the educational milieu. 

Copies of handouti) are included after the references so that readers may participate 
further in the ideas suggested by this research. This research project, as it continues, hopes 
to establish new criteria for the examination and use of visual materials. Teacher educators 
and teachers will promote a more visually literate audience with these activities. Evaluation 
of materials becomes one method by which to encourage a reexamination of teaching and 
learning from a more inclusive cultural dimension. 
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Instructional Materials Evaluation 



Title: 

Source, Date, Length, Cost: 

Subject Area/ Audience: 

Brief Description/Subject Matter: 



■ Language -.ysag^'. - ■ 


i'-bocliKot- 

Apply 


.. -Nottt ' 

■ -All. ' ' 
Poor 


-. jSk>me- 
What 


Compete ty 
NVcU . 


1 . Is the language used context specific? 










2. Will students understand the concepts 
presented, especially those presented within 
metaphors using familiar words in new ways? 










3. Is one point of view presented without the 
acknowledgement of other possibilities? 










4. Is the language used appropriate to the content 
but broad enough to allow other points of 
view? 










Comments: 


.Perspectiveis 










5. Is the framework for content selection 
explained? 










6. Is the design of the information obvious? Are 
designers' choices explained? 










7. Does the knowledge presented allow flexibility 
for student and teacher involvement? 










8. Is this material marketed in an easily usable 
and available format? 










9. Is material designed to reflect mainstream 
society's assumptions concerning relevant 
cultural features such as responsibility, 
education, career, family, religion, etc.? 
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Perspectives Contf ; : 


Apply 


:. AU "." 

Poor, 


Whil 


Well 


10. Does the material promote and reflect the 
distinctive character and values of all 
cultures? 










11. Is this material directed to the largest number 
of users to the exclusion of minority or 
smaller groups within the educational 
population? 










12. Is the value framework for this material 
evident? 










13. Are the students or teachers given the 

opportunity to question or explore the values 
framework used? 










14. Do the values presented interface with the 
values held by users? 










Comments: 


Curricular/Strategy Issues 










15. Is this material promoting effective and 
efficient learning to the exclusion of more 
broadly-based learning outcomeo? 










16. Is the underlying message of this material 
that technologically driven lemming is 
prefeited over other types of learning? 










17. Is the material appropriate educationally 
rather than selected as a result of some 
commercial effort? 










18. Is using this material the soundest solution to 
the educational problem involved? 










Comments: 



Derived from Rothe, J. P. (1991). Critical evaluation of educational software from a social 
perspective: Uncovering some hidden assumptions. In Hlynka, D. and Belland, J. 
(eds). Paradigms Regained^ New Jersey: Educational Technology Press, pp. 367-384. 

^ Nichols/Robinson/ Wiegmann. No use or reproduction without permission. 
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